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Facial modeling has been an ongoing research for many years and still 
shows research trend due to its relevance to current technology. Many 
applications incorporate facial expression modeling with the help of facial 
tracking, facial animation and facial recognition. The existing performance 
of the modeling method faces the challenges to perform well due to many 
factors. Currently, the use of 2D images and videos as inputs for modeling 
process are gaining popularity. However, current technologies and 
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1. INTRODUCTION 

Facial expression is an important aspect in non-verbal communication. Facial expression research 
not only addresses the areas of psychology and humanities, but it is also studied by computer science 
researchers. Machine learning, for example, has been used to recognize facial features and facial expressions, 
while in computer graphics facial expressions modeling add realism to animation, computer games and CGI 
films. Facial modeling is studied extensively in computer graphics since it provides the base of constructing 
different facial expressions and acts as a mechanism to analyze and synthesize facial expressions. It is 
commonly used for face recognition, human emotional analysis, drowsiness detection (such as in [1]), face 
tracking and 3D facial animation. 

Facial expression modeling provides the base of different facial expressions and construction 
mechanism to analyze and synthesize the facial expression. The process of modeling the facial modeling 
includes the face existence detection and implementation of face extraction mechanism such as face 
identification and expression from sequences of captured images. This modeling approach constructed 
standard form for different facial expression. Some of highlighted issues in facial modeling from 2D facial 
images are as follows: difficulties due to head pose, scale variation, pose scale, occlusion and incomplete 
geometric features. 3D scene that contains facial landmark can be used as reference for facial data in 
generating facial expression. However, the construction of 3D face model involves high level of difficulties 
in fundamental computer vision problem. 

Facial features and face parameters play important roles in the construction of 3D face model. Both 
facial features and facial parameter are needed to give detail appearance on the face model such as the 
position of the eyes, mouth and nose. The parameters also help in reflecting the emotional state and also 
describing the facial expression on the face model. Generally, facial expression modeling involves the use of 
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energy function. The energy function takes both poses of head and scales with consideration of landmark 
fitting to complement the traditional 2D facial expression reconstruction. Different energy optimization 
methods can be used to compute the counterpart of 3D facial expression from 2D facial images such as least 
square and gradient descent. The increasing level of energy function complexity due to degree of freedom 
such as identity, expression, head pose, and scale in fitting may lead to higher computation cost and 
probability of reaching local optimum that eventually limit the modeling result. Other than facial feature 
points, the images also require depth analysis since 3D model reconstruction requires depth to obtain the 
three dimensional view of the model. 

This paper presents a review on facial modeling. The following sections will discuss on related 
previous work involving few types of approaches including 2D-based modeling, 3D-based modeling 
(geometric, static and dynamic) and also deep learning which is the base of facial expression construction. In 
the next section, discussion on facial modeling approaches will be presented and finally, the conclusion will 
be given in last section. 


2. PREVIOUS WORK 

Facial modeling had been an on going research for many years. There were various works that have 
been presented by previous researchers. These modeling approaches are related to 2D-based modeling, 3D 
geometric based modeling, 3D static based modeling, 3D dynamic based modeling and modeling through 
deep learning which lead to better result of facial expression modeling. 


2.1. 2D-based modeling 

2D-based face model construction are based on 2D image features such as face contour, facial 
feature landmark and texture. Active shape model [2] and active appearance model [3] are classical model 
that were used to represent the facial feature by facial landmark while [4] improved the algorithm to reach 
out better analysis of facial expression. Other than that, facial feature can also be presented by facial contour 
[5]. The facial image analysis and facial expression clustering can be done by using face space that was 
constructed based on principal component analysis [6], [7]. The modeling process based on face space was 
able to help face detection [8]. 2D facial images usually include information of 2D data that can help in 3D 
estimation of shape, features layer (including landmark points) and occluding boundaries or texture edges 
called contour. The 3D morphable model (3DMM) fitting algorithm to 2D geometric information was also 
presented [9]. 


2.2. 3D Geometric based modeling 

Other than that, the construction of facial model also can be based on geometric features extraction 
from multi-facial images or 3D facial data. Facial model can be built using anthropometric facial features 
[10] and multi views of facial data [11]. Face modeling also can be done based on surface of B-Spline 
reconstruction [12] or sparse-iterative closest point [13]. Hybrid facial geometry algorithm (HFGA) can also 
be used without initialized model, as it can automatically extracted facial features and classifying the facial 
expression by the geometric values of facial features [14]. 


2.3. 3D Static based modeling 

The static facial feature space construction is one of the available methods for face modeling. A 
classical facial modeling method called 3D morphable model (3DMM) was used to reconstruct 3D facial data 
with special characteristic [15]. The 3DMM used fitting method and the method considers the edge 
information to improve the fitting speed [16]. The static modeling constructed the face space using facial 
feature analysis [17]. The different faces were transferred to regular representation and face space. The 
performance was limited since the method was depending to facial samples. For modeling method, more 
computations are required to ensure better result and performance. 


2.4. 3D Dynamic modeling 

A variety of facial samples with different expressions can also be used to construct facial model. 
There are past works that implemented the method. Deformation modeling was suggested to be applied for 
3D face matching [18]. Reconstruction of high fidelity 3D facial model using facial data which includes the 
texture, wrinkles, facial frontal and side images [19], [20]. Others suggested the usage of core tensor space to 
reduce computation of facial reconstruction, tensor subspace analysis for face modeling and face recognition 
[21], and local tensor for face recognition [22]. Multilinear models are used for facial expression modeling by 
[23] and also being constructed by Facewarehouse with facial surface fitting that was implemented for face 
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tracking and facial animation [24]. Reconstruction process usually needs the energy equation optimization 
but it will affect and limit the quality of facial modeling by local optimization. 


2.5. Deep learning 

One of the approaches in facial modeling is by using variation of facial features in deep learning and 
manifold learning. Multiple processing layers are applied in deep learning to learn data representation with 
different level of feature extraction [25] which eventually improves the performance such as The Deepface 
[26]; VGGFace [27] and FaceNet [28]. Some proposed methods for facial expression analysis such as 
restricted boltzmann machines [29] and deep fusion convolutional neural network for multi model are based 
on manifold framework [30]. Other than that, convolutional neural networks was also proposed for extended 
based method of facial expression to tackle data accuracy issues in machine learning [31]. Multi facial 
images deep neural network was suggested for facial expression recognition [32] as well as the 3D facial data 
flow that was mapped into grassman manifold [33] where combination of manifold learning and static facial 
modeling were introduced [34]. Shape space was also one of the features that can be used in this approach. 
Shape space can be used for facial data analysis [35] and elastic measure in shape space enables 3D faces 
differences to be analyzed [36]. Kendall shape space was proposed in reconstruction of 3D facial model in 
facial modeling for higher accuracy rate [37]. Taxonomy of state-of-the-art-image-base 3D object [38] 
through potraying the deep learning and manifold learning approach may lead to better result of facial 
expression recognition with complicated framework for facial expression modeling. 


3. DISCUSSION 

Each modeling method takes on a variety of challenges in producing the best performance. 3D 
reconstruction for 2D images is one of the fundamental issues of modeling and computer vision. By these 
previews of methods, the most preferable choice is reconstruction of the scene automatically by little or no 
user intervention. 3D scene reconstruction from 2D image is very challenging as 2D image contains less 
information from loss of depth information due to the nature of image forming process [39]. In order to 
obtain points of the scene, there are some image requirements to reconstruct 3D model: points projection of 
the scene on the image, matching ability to recognize and associate points, 3D coordinate determination 
based on association and calibration parameters, and obtaining mesh by finding surface own by the point to 
build model. 

There were various previous works that used 2D face modeling and 3D face modeling which 
include geometry modeling, static modeling and dynamic modeling. However, deep learning had become the 
current trend in solving problem on computer vision issues as it demonstrates good performance in 
identification, classification and target detection. Apart from that, it shows the ablity to lower the reliability 
on image preprocessing, feature extraction and shows better result compared to conventional approaches. 

Clearly, it is difficult to perform facial expression model due to head pose changes and facial 
changes based on expression. The occlusion and illumination that happened may cause information loss. 
Extracting the features as facial expression representation can be done by traditional methods but these 
approaches may not be able to handle it well due to the large pose variation. Pose normalization for mapping 
using facial features can be completed by exploiting the correlation of different poses by regression model in 
small face poses. Large face pose drop their performance due to accuracy of facial landmark detection. 3D 
geometrical transformation suggested by aligning 2D facial image with 3D model [40] tackled the 
normalization pose but is inconvenient for facial expression modeling. Most methods learn multiple 
classifiers for each pose which eventually will be time-consuming. Recovering 3D coordinate of an image 
using features matching across images based on a variety of view and 3D construction by segmented 2D 
silhouettes of the same object with multiple images that captured by well-calibrated cameras. Quality of 3D 
model made by these models is sensible but not feasible in many situations. 

Comparison made by [41] which is made up of illumination changes, head motion, aging, facial 
make-up, acquisition of data, availability of data, computational cost, measurement of face surface, action 
units (AU)s low-intensity performance, acquisition and recognition, neutral scanning, AUs and facial sample 
database availability and real-time basically shows the pro and cons of using 2D and 3D. However, both 2D 
and 3D are having also similar problems such as lack of spontaneous dataset, manually marked features and 
facial points, and equal accuracy needed for all expression. 

Recently, several researchers revealed the deep learning in computer vision and visual task inspired 
by deep network where they managed to get good result in identifying, classifying, and detecting. Deep 
convolutional neural network (CNN) leads in processing images, videos, speech and tools while current NN 
is leaning more towards the sequential data like text and speech where both success rate are depending on 
input data [42]. 3D database in CNN architecture for training data will affect the rate of performance. 
Generative adversarial network (GAN) is also a popular model due to its ability through the adversarial 
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training with the ability to produce sharp images and is widely used for face-related tasks. This encourages 
the usage of GAN into the training set. Other than that, accuracy in deep learning can be boosted up using 
comprehensive parameter turning as the result of facial expression based on deep learning that utilizes the 
transfer learning of available models with results of more than 70% accurate rate [43]. 


4. CONCLUSION 

This paper focuses on methods of facial expression modeling. Reconstructing 3D facial expression 
based on 2D face modeling also deals with similar problems as in the area of static and dynamic modeling, 
geometric modeling and deep learning. Based on preview of previous work, deep learning shows 
improvement of result based on accuracy compared to other method of facial modeling. 3D facial expression 
modeling accuracy rate is influenced by the shape representation; either single image, multiple images or 
sequences of images. Architecture of network implies the output representation and eventually affected the 
computational efficiency and reconstruction quality. Training mechanism applied include variation of 
datasets, loss functions that can affect the output quality and training procedure. 

Based on current review, deep learning method are showing the high percentage of accuracy result 
for facial model however the process is mainly time-consuming. Facial modeling using deep learning method 
with the enhancement of time consumption due to work load process would be a recommendation for future 
research area. 
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